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EXECUTIVE  SUMMARY 


1.  BACKGROUND.  A reliable  system  to  control  the  quality  of  procurement 
instruments  and  provide  useful  management  information  regarding  document 
quality  is  needed  within  the  Army  Materiel  Development  and  Readiness  Command 
(DARCOM).  Therefore,  the  Army  Procurement  Research  Office  was  tasked  to 
develop  a quality  improvement  system  for  procurement  instruments  that 
satisfies  both  requirements. 

2.  STUDY  OBJECTIVES.  The  objectives  of  this  study  are  to  develop  and  to 
test  a system  that  (i ) aids  in  the  improvement  and  control  of  the  quality  of 
DARCOM  procurement  instruments  and  (ii)  provides  useful  management  informa- 
tion compatible  with  the  DARCOM  procurement  management  program. 

3.  STUDY  APPROACH.  The  approach  taken  to  achieve  these  objectives  includes 
the  identi fi cation  and  evaluation  of  quality  indicators,  a review  and 
analysis  of  existing  software  quality  control  systems  and  techniques,  inter- 
views with  procurement  personnel,  and  the  synthesis  and  test  of  a system 
that  meets  DARCOM' s requirements. 

4.  SUMMARY  AND  CONCLUSIONS.  The  Quality  Improvement  System  for  Procurement 
Instruments  (QISPl)  as  presented  in  this  report  has  considerable  potential 
as  a tool  for  procurement  managers  to  control  the  quality  of  their  instru- 
ments. Document  quality  measurement  and  control  begins  with  a quality  indi- 
cator (QI)  calculation  based  upon  the  number  and  type  of  instrument  defi- 
ciencies per  page.  Standard  control  charts  are  used  to  determine  document 
acceptability  and  track  quality  levels  at  the  commodity  commands.  The  QI 
values  from  each  cormiand  are  adjusted  at  the  HQ  DARCOM  level  to  reflect  review 
board  proficiency  and  sampling  percentages  which  results  in  the  final  per- 
formance indicator  (PI)  calculation  for  all  of  DARCOM.  System  test  results 
from  tests  conducted  at  selected  major  procurement  activities  of  DARCOM  will 

be  carefully  evaluated  during  and  after  the  test  to  confirm  the  appropriateness 
and  utility  of  the  system  and  to  make  any  necessary  improvements.  Recommenda- 
tions regarding  QISPl  implementation  are  postponed  until  the  tests  are  completed. 


r*cttssi9»  tv 


1 ITU 

White 

SfCtlM  fc,  J 

05C 

luff  SectiDi  /fa  | 

UimOUlCFB 
JUSTIflClTIJ*  

a i 

1 

i 

ir 

CISTRIIUTIVH  IIUlHll  !’  CnOEl  1 

tut. 

AVAIL,  ini 

* snail  " 

TABLE  OF  CONTENTS 


PAGE 

EXECUTIVE  SUMMARY ii 

TABLE  OF  CONTENTS iii 

LIST  OF  FIGURES v 

LIST  OF  GRAPHS vi 

LIST  OF  TABLES vii 

CHAPTER 

I.  INTRODUCTION  1 

A.  Background/Problem  1 

B.  Study  Objectives  2 

C.  Study  Approach  3 

D.  Report  Organization 3 

II.  QUALITY  IMPROVEMENT  SYSTEM  DEVELOPMENT  4 

A.  Quality  Indicators  4 

B.  Quality  Control  Techniques  6 

C.  Quality  Control  Systems. 8 

D.  Quality  Improvement  System  for  Procurement  Instruments  ....  11 

III.  QUALITY  IMPROVEMENT  SYSTEM  OPERATION  21 

A.  Operation  at  Commodity  Commands 21 

B.  Operation  at  HQ,  DARCOM 27 

IV.  QUALITY  IMPROVEMENT  SYSTEM  TEST 32 

A.  Computer  Simulation 33 

B.  Test  Applications 43 


LIST  OF  FIGURES 


PAGE 

FIGURE 

1.  PROCUREMENT  QUALITY  CHECKLIST 22 

2.  EXAMPLE  CONTROL  CHART  FOR  CONTRACT  QI 25 

3.  VIEWGRAPH  FORMAT  FOR  PI  SUMMARY 30 


v 


LIST  OF  GRAPHS 


GRAPH 


PAGE 


1.  FREQUENCY  DISTRIBUTION  OF  QI  VALUES  WITH  MAJOR/MINOR 

DEFICIENCIES  = .05,  .10 34 

2.  FREQUENCY  DISTRIBUTION  OF  QI  VALUES  WITH  MAJOR/MINOR 

DEFICIENCIES  = .10,  .20 36 

3.  FREQUENCY  DISTRIBUTION  OF  QI  VALUES  WITH  MAJOR/MINOR 

DEFICIENCIES  = .005,  .01  37 

4.  FREQUENCY  DISTRIBUTION  OF  QI  VALUES  WITH  MAJOR/MINOR 

DEFICIENCIES  = .05,  .10 38 

5.  FREQUENCY  DISTRIBUTION  OF  QI  VALUES  WITH  MAJOR/MINOR 

DEFICIENCIES  = .05,  .10 39 

6.  FREQUENCY  DISTRIBUTION  OF  QI  VALUES  WITH  MAJOR/MINOR 

DEFICIENCIES  = .05,  .10 40 


vi 


LIST  OF  TABLES 


TABLE 

1.  SENSITIVITY  OF  QI  TO  NUMBER  OF  PAGES 

2.  SIMULATED  PI  VALUES 


CHAPTER  I 


INTRODUCTION 

A.  BACKGROUND/PROBLEM 

All  commodity  commands  within  the  US  Army  Materiel  Development  and 
Readiness  Command  (DARCOM)  are  concerned  with  the  quality  of  their  procure- 
ment instruments.  However,  none  really  knows  just  how  good  or  how  bad  their 
instruments  are;  nor  do  they  know  what  their  procurement  document  quality 
trend  is.  This  is  primarily  because  the  commands  have  no  formal 
standard  to  measure  their  documents  against  and  therefore  no  meaningful 
management  information  on  document  quality.  Without  such  management  informa- 
tion, procurement  instrument  quality  control  and  improvement  are  at  best 
haphazard.  The  lack  of  the  information  also  makes  it  difficult  to  state  with 
any  confidence  whether  the  present  quality  of  DARCOM  procurements  is  acceptable 
or  unacceptable. 

To  remedy  this  situation  a procurement  instrument  quality  control  system 
is  needed.  The  system  would  not  only  assure  procurement  management  that 
their  documents  are  of  acceptable  quality  but  also  provide  the  document 
quality  information  that  is  presently  lacking  in  DARCOM.  A previous  Army 
Procurement  Research  Office  (APRO)  report  recognized  the  need  for  such  a 
quality  control  system  and  recommended  that  one  be  designed  and  implemented 
in  DARCOM  in  conjunction  with  permanent,  full-time  review  boards J 


Nick,  R.  W.  et  al.  "Analysis  of  AMC  Solicitation  and  Award  Review 
Procedures."  Arrqy  Procurement  Research  Office,  Fort  Lee,  Virginia  23801. 
September  1975. 
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Additionally,  any  system  developed  to  improve  the  quality  of  DARCOM 

procurement  instruments  must  also  be  compatible  with  the  DARCOM  Procurement 

2 

Management  System  (PROMS)  that  plans  and  controls  its  procurement  program. 

The  document  quality  information  to  be  provided  by  this  quality  control  system 
must  satisfy  procurement  management  requirements  at  both  the  commodity  command 
level  and  the  DARCOM  HQ  level.  To  meet  these  requirements,  the  Procurement 
and  Production  Directorate,  DARCOM,  tasked  APRO  with  a project  to  develop  and 
test  a quality  improvement  system  for  procurement  instruments. 

B.  OBJECTIVES 

The  general  objective  of  the  project  is  to  develop  and  to  test  a system 
that  (i)  aids  in  the  improvement  and  control  of  the  quality  of  DARCOM  procure- 
ment instruments  and  (ii)  provides  useful  management  information  compatible 
with  the  DARCOM  procurement  management  program.  Specific  objectives  en- 
compassed by  this  two-fold  objective  include: 

1.  Establish  a measurement  baseline  or  criteria  as  to  what  constitutes 
an  acceptable  procurement  document. 

2.  Synthesize  a system  using  available  statistical  quality  control 
techniques. 


2 

Williams,  R.  F.  et.  al.  "Procurement  Goal  Setting  and  Performance 
Analysis."  Army  Procurement  Research  Office,  Fort  Lee,  Virginia  23801. 
July  1975. 
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3.  Define  a performance  ir.dex  for  procurement  document  quality  and 
the  tasks,  targets,  and  methods  of  analysis  required  as  input  to  PROMS. 


4.  Test  the  system  and  make  modifications  as  required. 

C.  STUDY  APPROACH 

To  achieve  these  objectives  the  following  study  approach  is  taken: 

1.  Identify  factors  that  affect  or  indicate  procurement  instrument 
qua  1 ity. 

2.  Review  and  analyze  literature  on  existing  software  quality  control 
systems  and  techniques  for  possible  application  to  DARCOM  procurement 
instruments . 

3.  Interview  quality  control  and  procurement  personnel  experienced  in 
this  area. 

4.  Synthesize  a proposed  system  that  meets  the  requirements  stated  in 
the  objectives  and  test  it  at  at  least  one  commodity  command. 

5.  After  any  required  modifications  are  made,  submit  recommendations 
for  implementation  to  DARCOM  Headquarters. 

This  report  presents  the  results  of  APRO's  research  effort 
thus  far.  A final  report  will  be  published  following  the  system  tests. 

D.  REPORT  ORGANIZATION 

Chapter  II  discusses  existing  quality  control  systems  and  techniques 
and  presents  the  quality  improvement  system  developed  for  use  in  DARCOM. 
Chapter  III  describes  the  envisioned  operation  of  the  system  at  both  the 
commodity  command  level  and  the  DARCOM  HQ  level.  Two  system  test  approaches 
are  discussed  in  Chapter  IV,  and  Chapter  V briefly  concludes  this  report. 
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CHAPTER  II 


QUALITY  IMPROVEMENT  SYSTEM  DEVELOPMENT 

The  primary  mission  of  the  Procurement  and  Production  Directorates  at  the 
DARCOM  commodity  commands  is  to  make  "good"  procurements.  A fundamental 
assumption  of  this  report  is  that  high  quality  instruments  will  result  in  good 
procurements;  if  the  instrument  quality  is  controlled,  the  quality  of 
the  procurements  will  likewise  be  controlled.  Although  a good  Instrument 
will  not  guarantee  a good  procurement,  it  does  increase  the  probability  of 
one  since  it  reflects  the  quality  of  the  effort  expended  in  planning  for  and 
executing  the  procurement. 

A.  QUALITY  INDICATORS 

The  basic  function  of  a procurement  instrument  is  to  legally,  and 
within  the  confines  of  established  general  procurement  policy,  obtain  the 
required  goods  or  services  at  the  required  time  and  place  and  at  a fair  and 
reasonable  price.  The  first  step  in  developing  a quality  control  system  for 
procurement  instruments  is  to  establish  a quality  indicator,  something 
that  indicates  how  well  the  instrument  satisfies  this  function.  Once  the 
indicator  is  chosen,  an  appropriate  quality  control  system  can  be  designed 
around  it. 

A number  of  potential  quality  indicators  were  considered  in  the 
search  for  something  that  directly  reflected  instrument  quality  and  was 
measurable  and  controllable  by  procurement  personnel.  They  are  summarized 
in  the  following  list: 


1.  Number  of  procurement  solicitation  cancellations. 

2.  Number  of  solicitation  amendments. 

3.  Number  of  supplemental  agreements. 

4.  Number  of  bid  protests  and  contract  appeals  upheld. 

5.  Customer  satisfaction  with  good/service. 

6.  Number  and  length  of  procurement  reviews. 

7.  Number  and  type  of  instrument  deficiencies. 

The  problem  with  most  of  these  variables  as  quality  indicators  is  that 
they  are  not  reliable;  that  is,  they  do  not  consistently  and  accurately  re- 
flect instrument  quality.  Since  difficulties  such  as  solicitation  amendments, 
contract  change  orders,  or  protests  and  appeals  can  result  from  good  docu- 
ments and  through  no  fault  of  procurement  personnel,  these  variables  are 
useful  only  as  rough  measures  of  good  procurements.  Additionally,  such 
difficulties  will  not  necessarily  follow  from  a low  quality  instrument.  A 
competent  contractor  with  a low  quality  instrument  can  still  yield  a good 
procurement.  Unlike  most  hardware  quality  measurements  which  are  accurate 
and  precise,  procurement  instrument  quality  measurements  are  inherently 
subjective  and  imprecise.  Therefore,  no  direct,  exact  indicator  was  found 
for  document  quality. 

Although  none  of  the  variables  listed  really  met  what  was  desired  in  a 
quality  indicator,  the  number  and  type  of  instrument  deficiencies  came  the 
closest  to  satisfying  the  requi rements.  The  number  and  type  of  deficiencies 
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detected  during  review  by  a procurement  review  board  is  a direct  measure- 
ment of  document  quality.  It  is  reliable  to  the  extent  that  the  review 
board  is  thorough  and  consistent  in  Its  review.  Document  quality  as  measured 
by  the  number  and  type  of  deficiencies  can  be  controlled  by  the  procurement 
personnel  preparing  and  reviewing  the  documents.  A more  detailed  discussion 
of  this  quality  indicator  is  presented  in  Section  D. 

After  the  appropriate  quality  indicator  was  established,  an  investigation 
was  made  of  available  quality  control  techniques  and  systems. 

B.  QUALITY  CONTROL  TECHNIQUES 

There  are  a number  of  quality  control  techniques  that  have  potential 
application  to  procurement  instruments.  Rating  schemes  ranging  from  color 
coding  to  letter  grading  have  been  tried  and  offer  some  usefulness.  Various 
statistical  quality  control  techniques  offer  assurance  of  the  desired  level 
of  quality  at  the  lowest  overall  cost.  Of  the  many  techniques  examined,  twc 
statistical  techniques,  the  "C"  chart  for  defects  per  unit  and  sampling  for 
inspection  by  attributes,  appeared  to  have  the  greatest  potential  for  appli- 
cation within  DARCOM. 

The  "C"  chart  is  a control  chart  for  attributes  that  shows  the  count  of 
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defects  following  inspection  of. the  item  of  concern.  The  upper  and  lower 
control  limits  for  the  "C"  chart  are  based  on  the  Poisson  probability 


^Grant,  E.  L.  and  Leavenworth,  R.  $.  Statistical  Quality  Control, 
McGraw-Hill  Book  Company,  1972.  Fourth  Edition. 


distribution.  Since  opportunities  for  defects  are  numerous  in  procurement 
instruments  but  the  chances  of  a defect  occuring  in  any  one  place  are  small, 
the  Poisson  distribution  is  a reasonable  assumption.  The  "C"  chart  has 
application  to  100  percent  inspection  or  sampling  and  can  be  adapted  to 
quality  racing  schemes  based  on  demerits  per  unit. 

The  purpose  of  a control  chart  for  procurement  instruments  would  be  one 
or  more  of  the  following: 

1.  To  discover  the  average  quality  level  of  documents  reviewed  over 
a period  of  time. 

2.  To  bring  to  management's  attention  any  changes  in  the  average 
qual i ty  level . 

3.  To  discover  those  out-of-control  high  points  that  require  action  to 
identify  and  correct  causes  of  bad  quality. 

4.  To  discover  those  out-of-control  low  points  that  indicate  either 
relaxed  inspection  standards  or  erratic  causes  of  quality  improvement  which 
might  be  converted  into  causes  of  consistent  quality  improvement. 

Sampling  for  inspection  by  attributes  is  another  statistical  tool 
applicable  to  procurement  instruments.  Briefly,  sampling  for  inspection  by 
attributes  involves  taking  a portion  (sample)  from  the  total  product  available 
(lot)  and  inspecting  that  portion  (counting  the  number  of  defects)  to 
determine  if  the  total  is  acceptable  with  respect  to  a given  requirement. 

MIL- STD- 1 05D , which  is  referred  to  in  most  government  prodjct  specifications. 
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gives  a more  thorough  discussion  of  sampling  for  inspection  by  attributes. 

While  it  is  normally  used  for  hardware  inspection,  the  principles  can  be 
applied  to  software  inspection  including  procurement  instruments.  Sampling 
plans  other  than  those  specified  in  MIL-STD-105D  are  also  available  for 
consideration. 

C.  PRESENT  QUALITY  CONTROL  SYSTEMS 

While  none  of  these  techniques  is  currently  being  applied  in  DARCOM, 
all  comodity  commands  have  developed  checklists  to  aid  in  controlling  pro- 
curement instrument  quality.  These  checklists  are  used  by  the  procuring  con- 
tracting officer  (PCO)  when  he  prepares  the  action  to  assure  that  it  is  complete 
and  accurate.  The  checklists  are  similarly  used  by  a procurement  reviewer 
when  he  reviews  the  actions.  Checklists  have  a definite  shortcoming  in  that 
they  do  not  provide  a quantitative  measurement  of  the  quality  level  of  the 
procurement  package. 

Most  of  the  activities  contacted  outside  of  DARCOM  also  use  some  type  of 
document  checklist  to  control  the  quality  of  their  procurements;  however,  a 
few  DOD  organizations  have  included  the  checklist  as  part  of  a larger  system. 

The  Air  Force  and  Defense  Supply  Agency  have  both  developed  quality  control 
systems  based  on  statistical  techniques  and  are  using  them  to  control  the 
quality  of  their  procurement  actions. 


4 

"Sampling  Procedures  and  Tables  for  Inspection  by  Attributes," 
MIL-STD-105D,  US  Government  Printing  Office,  Washington,  D.C.  20402, 
29  April  1963. 
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Specifically,  one  Air  FOrce  Logistic  Center  (ALC)  has  developed  and 
implemented  a document  quality  assurance  program  that  includes  a combination 
of  checklists,  inspection  sheets,  and  judgmental  Inspection  of  non-checklist 
items  in  the  procurement  package.  A review  board  records  the  defects  (either 
major  or  minor)  found  during  the  review  process  and  returns  these  recorded 
defects  along  with  the  reviewed  actions  to  the  buyer  and  PCO  for  follow-up. 
Monthly  simmaries  that  show  quality  performance  and  trends  are  prepared  for 
each  buying  division  and  are  given  to  the  division  management  for  their  in- 
formation and  corrective  action.  The  ALC  reviews  100  percent  of  the  actions 
equal  to  or  greater  than  specific  dollar  thresholds,  which  depend  upon  the 
type  of  procurement,  but  it  reviews  only  a sample  of  actions  less  than  these 
thresholds.  A sampling  system  called  Procurement  Integrated  Quality  Assurance 
System  (PIQAS)  is  employed  which  uses  the  techniques  in  MIL-STD-105D  to  pro- 
vide a practical  means  of  assessing  the  quality  of  smaller  dollar  value  pro- 
curements. The  total  number  of  defects  found  during  the  review  of  the 
sampled  documents  is  compared  with  the  acceptance  number  (Ac)  stated  in 
MIL-STD-105D  for  the  present  sample  size  and  desired  Acceptable  Quality  Level 
(AQL).  The  results,  which  show  lot  acceptability,  are  summarized  quarterly 
and  sent  to  each  buying  division  for  their  information. 

The  ALC  feels  that  their  overall  quality  program  (1)  is  effective  in 
identifying  problems,  (2)  is  operated  with  optimum  expenditure  of  resources 
for  the  desired  results,  and  (3)  indicates  that  procurement  action  quality 
is  within  limits  of  acceptabi 1 ity.  The  claimed  benefits  include  improved 
procurement  quality,  informed  personnel,  a practical  training  vehicle,  and  a 
measurement  of  personnel  effectiveness. 
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A second  ALC  has  developed  a system  similar  to  PIQAS  called  Program  for 
Improvement  of  Contract  Actions  (PICA).  Basically,  PICA  is  a sampling  for 
inspection  by  attributes  system  that  includes  a statistical  analysis  of 
quality  performance  based  on  the  number  of  procurement  actions  and  error  rate 
for  each  sample  stratification.  Stratifications  are  by  buying  section  and 
PCO.  Confidence  statements  can  be  made  regarding  document  quality  and  the 
source  of  error  can  be  pinpointed.  Another  approach  under  consideration  at 
this  ALC  is  to  use  the  F-distribution  to  test  for  significant  differences 
among  sections  and  among  PCOs. 

The  Defense  Supply  Agency  has  developed  a System  for  Procurement  Evalua- 

5 

tion.  Quality  Assessment  and  Reporting  (SPEAR).  Similar  to  the  Air 
Force  systems  in  that  SPEAR  is  a sampling  system,  it  uses  MIL-STD-105D  to 
determine  sample  sizes  and  acceptance  numbers.  The  SPEAR  uses  small  purchase 
and  large  purchase  checklists  to  guide  document  review  and  determines  the 
sample  size  according  to  the  dollar  value  of  the  procurement  action.  It 
also  classifies  defects  into  three  groups:  critical,  major,  and  minor.  The 

percent  defective  or  defects  per  100  units  is  calculated  from  the  sample 
as  the  performance  indicator  for  each  procurement  action  lot.  Monthly  re- 
ports are  prepared  which  summarize  the  performance  achieved. 


5 

Gorski,  W.  B.  "A  System  for  Procurement  Evaluation,  Quality 
Assessment  and  Keporting."  Contract  Review  Office,  Defense  General  Supply 
Center,  Richmond,  Virginia.  1970. 
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The  quality  index  for  drawing  packages,  a non-government  system  for 
software  quality  control,  was  also  reviewed.6  In  this  system  the  number  of 
errors  are  counted  on  each  drawing  during  the  review  of  a drawing  package 
and  a quality  index  is  calculated  using  a negative  exponential  function.  A 
distinction  is  made  between  major  and  minor  errors.  The  calculated  quality 
index  is  compared  with  historical  standards  for  acceptability.  Provision  is 
also  made  in  the  system  for  package  sampling,  package  complexity  and  error 
correction.  A final  quality  index  measurement  is  obtained  after  allowance 
is  made  for  each  of  these  items. 

D.  QUALITY  IMPROVEMENT  SYSTEM 

After  a thorough  review  of  the  available  quality  control  techniques  and 
systems,  the  control  chart  approach  was  chosen.  The  reasons  for  this  choice 
follow. 

Sampling  for  inspection  by  attributes  provides  information  regarding 
general  quality  and  which  instrument  attributes  are  problem  areas,  but  it 
shows  little  about  individual  instrument  quality.  For  example,  assuming 
each  checklist  line  item  is  an  attribute,  a review  of  a sample  of  10  con- 
tracts might  show  only  one  deficiency  in  each  line  item  which  overall  could 


6Sargon,  V.  and  Murphy,  J.  "A  Quality  Index  for  Drawing  Packages," 
Graphic  Science,  New  York,  New  York  10016.  March  1971,  pages  16-21. 
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be  an  acceptable  quality  level.  But  if  that  one  deficiency  in  each  line 
item  was  contained  in  one  or  two  contracts,  those  contracts  would  be  of 
unacceptable  quality  but  would  be  considered  acceptable  under  the  sampling 
system. 

Control  charts  for  attributes  provide  information  regarding  general 
quality  levels  and  individual  instrument  quality,  but  by  itself,  a chart 
shows  nothing  about  which  areas  of  the  documents  are  problem  areas.  That 
must  be  determined  through  an  independent  review  of  the  documents. 

Additionally,  large  dollar  volume  instruments  cannot  be  sampled  by  the 
commodity  commands  but  must  be  100  percent  reviewed,  at  least  to  begin  with, 
and  it  is  not  feasible  to  review  all  small  dollar  value  documents.  There- 
fore, two  separate  systems  would  be  required  to  accomnodate  this  sampling 
situation.  Since  individual  instrument  quality  was  considered  to  be  more 
important  than  attribute  quality  and  only  one  simple  approach  was  desired, 
the  control  chart  approach  was  selected. 

The  system  structure  was  constructed  by  using  the  number  and  type  of 
instrument  deficiencies  as  the  quality  indicator  and  the  control  chart  as 
the  general  approach.  The  "C"  chart  was  a logical  starting  point. 

The  standard  "C"  chart  is  appropriate  where  articles  or  subgroups  of 
articles  have  substantially  equal  opportunity  for  the  occurrence  of  defects. 
The  measured  variable,  C,  is  the  number  of  defects  or  deficiencies  observed 
during  instrument  review.  It  might  be  possible  to  assume  that  each 
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procurement  instrument  has  substantially  the  same  opportunity  for  the 
occurrence  of  deficiencies,  but  it  would  be  a liberal  assumption  since  the 
instruments  vary  so  in  number  of  pages  and  complexity. 

An  additional  complication  is  that  some  deficiencies  are  more  serious 
than  others.  This  was  recognized  early  and  is  considered  by  the  quality 
indicator  in  the  type  of  instrument  deficiencies.  It  does  not  present  a 
problem  for  the  "C"  chart  since  an  adaptation  based  upon  demerits  per  unit 
rather  than  defects  per  unit  is  available.  Demerits  of  different  numerical 
value  can  be  assigned  to  each  class  of  deficiency,  and  the  total  weighted 
deficiencies  (demerits)  per  unit  can  be  plotted  on  a control  chart  just  as 
in  the  basic  "C"  chart.  However,  acceptance  of  the  "C"  chart  adaptation 
based  on  demerits  per  unit  was  reserved  until  other  possibilities  were 
exhausted. 

Since  no  other  existing  technique  suited  the  specific  requirements  of 
this  quality  control  system  for  procurement  instruments,  a return  to  funda- 
mental statistics  was  necessary. 

1.  Quality  Indicator 

The  quality  indicator  (QI)  to  be  controlled  is  the  number  and  type 
of  deficiencies  in  a procurement  instrument.  Since  the  complete  instrument 
varies  so  in  number  of  pages  and  complexity,  a different  unit  of  measurement 
is  needed.  The  individual  instrument  page  is  an  alternative.  It  is  reasonable 
to  assume  that  the  individual  pages  are  statistically  independent  from  each 
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other  and  have  substantially  equal  opportunity  for  the  occurrence  of  errors. 
This  is  "more  reasonable"  than  assuming  the  same  for  the  complete  instrument. 
Also,  to  begin  with,  only  two  types  of  deficiencies  will  be  recognized  - 
major  and  minor.  Using  this  assumption  and  deficiency  classification,  the 
specific  expression  for  the  quality  indicator  can  be  developed. 

The  number  of  major  deficiencies  (A),  and  the  number  of  minor  deficiencies 
(B),  on  each  document  page  are  two  independent  random  variables  that  can  be 
described  by  some  probability  distribution  (most  likely  the  Poisson),  but 
the  distribution  may  not  matter  as  will  be  seen  later.  A third  factor,  C, 
is  introduced  as  the  weighting  constant  for  the  major  deficiencies.  If  all 
the  deficiencies  on  each  page  of  the  instrument  are  totaled,  the  QI  is  simply 
the  sum  of  the  B values  plus  the  A values  weighted  by  C as  expressed  in 
equation  (1). 

n n 

QI  *£  Bi  + c ' Z Ai  (') 

i=l  i-1 

where: 

QI  is  the  quality  indicator, 

A is  the  number  of  major  deficiencies, 

B is  the  number  of  minor  deficiencies, 

C is  the  weighting  constant, 
n is  the  number  of  instrument  pages. 


A problem  with  using  equation  (1)  as  the  quality  indicator  is  that  the 

probability  distribution  of  QI  is  unknown;  this  would  not  allow  the  use  of 

parametric  statistics.  But,  if  both  summations  on  the  right-hand  side  of 

the  equation  are  divided  by  n,  the  resulting  summation  of  means  tends  toward 

normality  because  of  the  central  limit  theorem. 

"If  the  individual  random  variables  going  into  the  sample 
are  not  normally  distributed,  but  their  number  is 
sufficiently  large,  then  the  central  limit  theorem  indi- 
cates that  the  sample  mean  tends  toward  normality,  and 
the  distribution  theory  for  normally  distributed  random 
variables  is  relevant. 

Since  the  number  of  pages  in  a procurement  instrument  usually  falls 
between  30  and  100  and  averages  approximately  60,  the  "sufficiently  large" 
criterion  should  be  met.  Also  the  expected  deficiency  occurrence  rate 
should  be  large  enough  to  allow  the  normal  approximation.  A further  dis- 
cussion of  these  assumptions  and  their  validity  is  presented  in  Chapter  IV. 

The  sum  of  independent,  normally  distributed  random  variables  is  also 
normally  distributed  regardless  of  the  number  of  variables  in  the  sum,  and 
constants  do  not  affect  the  distribution's  normality.  Therefore,  a further 
breakout  of  deficiency  type  into  more  than  two  classifications  is  possible 
in  the  future  if  desired.  The  resulting  expression  for  a normally  distri- 
buted quality  indicator  is  given  in  equation  (2). 


^Bowker,  A.  H.  and  Lieberman,  G.  J.  Engineering  Statistics,  Prentice- 
Hall,  Inc.,  Englewood  Cliffs,  New  Jersey,  1959.  Page  66. 
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where : 

QI  is  the  weighted  number  of  deficiencies/page, 

B is  the  number  of  minor  deficiencies, 

A is  the  number  of  major  deficiencies, 

C is  the  weighting  constant, 
n is  the  number  of  instrument  pages. 

This  quality  indicator  expressed  in  deficiencies  per  page  is  theoretically 
correct,  easily  understood,  and  directly  reflects  procurement  instrument 
quality.  Therefore,  it  was  chosen  over  the  "C"  chart  adaptation  based  upon 
demerits  per  unit. 

It  mav  not  be  necessary  to  resort  to  distributional  approximation  for 


the  QI  in  equation  (2).  The  exact  distribution  of  this  QI  was  determined 
8 


to  be  chi-square.  If  the  number  of  occurrences  of  major  and  minor  deficien- 
cies on  any  page  are  assumed  to  each  follow  a Poisson  distribution,  the 
resultant  QI  follows  a chi-square  distribution.  Therefore,  the  exact 
percentage  points  for  control  limits  can  be  readily  calculated  using  chi-square 


8r 


Solomon,  Herbert.  Unpublished  memo  dated  1 Sep  76  from  Professor 
Solomon,  Stanford  University,  to  APRO  regarding  this  distributional 
problem. 
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tables.  As  the  deficiency  parameter  and  the  number  of  instrument  pages 
increase,  the  chi-square  tends  toward  normality.  The  decision  as  to  which 
approach  is  better  for  this  system;  i.e.,  the  normal  approximation  or  the 
exact  chi-square,  is  delayed  until  after  the  test  application  is  complete 
as  discussed  in  Chapter  IV. 

Once  the  quality  indicator  is  established  and  its  distribution  known, 
the  control  chart  can  be  structured  accordingly.  An  example  control  chart 
for  the  Quality  Improvement  System  for  Procurement  Instruments  (QISPI)  is 
presented  in  Figure  2 of  Chapter  III. 

Up  to  this  point  in  the  development  of  the  QISPI,  only  those  deficiencies 
detected  during  document  review  have  been  considered.  But  to  determine  the 
overall  quality  of  the  procurement  instruments,  the  deficiencies  that  go 
undetected  during  instrument  preparation  and  review  must  also  be  considered. 
The  performance  indicator  (PI)  concept  accounts  for  not  only  the  undetected 
deficiencies  but  also  the  number  of  documents  sampled  versus  100  percent 
reviewed.  Whereas  the  QI  values  are  measures  of  incoming  quality,  the  PI 
values  become  measures  of  outgoing  quality. 

2.  Performance  Indicator  (PI). 

It  is  impractical  to  attempt  to  remove  through  inspection  all  errors 
or  deficiencies  in  a procurement  instrument.  Therefore,  an  unknown  number 
of  deficiencies  will  remain  undetected  in  each  document  throughout  prepara- 
tion and  review.  In  order  to  estimate  the  total  number  of  deficiencies 
remaining  in  the  instrument  from  the  number  detected,  consideration  must  be 
given  to  the  proficiency  of  the  review  board  in  detecting  errors.  Review 
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board  proficiency  is  a function  of  many  factors  including  instrument  com- 
plexity and  time  allowed  for  review.  Since  it  is  not  possible  to  accurately 
quantify  and  consider  all  factors,  a simple  scale  adjustment  to  the  quality 
indicator  is  made  which  yields  the  performance  indicator  in  equation  (3). 


where: 

PI  is  the  performance  indicator  in  deficiencies/page, 

QI  is  the  quality  indicator  in  deficiencies/page, 

PF  is  the  review  board  proficiency  factor  (0  < PF  £ 1.0). 

A PF  equal  to  one  is  unlikely  since  it  would  indicate  that  all  errors 
present  were  detected  and  corrected.  Since  all  deficiencies  detected  during 
review  are  corrected  by  the  preparer  before  solicitation  issuance  or  con- 
tract award,  this  PI  becomes  a measure  of  the  weighted  number  of  defi- 
ciencies remaining  in  the  instrument  after  review.  It  is  appropriate  for 
those  documents  that  are  reviewed  100  percent.  For  that  group  of  documents 
of  which  only  a portion  is  reviewed,  an  additional  adjustment  is  necessary. 

To  account  for  the  deficiencies  in  those  instruments  prepared  but  not 
reviewed,  a second  factor  is  introduced  called  the  sampling  factor  (SF). 

SF  indicates  the  percent  of  the  group  reviewed.  The  resultant  expression 
for  the  PI  for  a sampled  group  is  given  in  equation  (4). 
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(4) 


(QI ) (SF) 

where: 

PI  is  the  performance  indicator  for  the  group  in  deficiencies/page, 

QI  is  the  average  quality  indicator  of  the  sample  in  deficiencies/page, 
PF  is  the  review  board  proficiency  factor  (0  < PF  <_  1.0), 

SF  is  the  sampling  factor  (0  < SF  <_  1.0). 

The  quality  level  of  the  entire  group  is  estimated  to  be  equal  to  the  average 
quality  level  of  the  sample;  therefore,  the  larger  the  sample,  the  better 
the  group  estimate.  An  SF  of  zero  is  not  allowed  since  that  would  indicate 
zero  percent  of  the  group  was  reviewed,  and  an  estimate  based  on  a sample 
size  of  zero  is  not  statistically  possible.  An  SF  of  one  reduces  equation 
(4)  to  equation  (3)  and  indicates  100  percent  of  the  group  was  reviewed  and 
corrected. 

By  combining  equations  (3)  and  (4)  and  weighting  each  factor  by  the 
applicable  number  of  documents,  the  final  expression  for  PI  is  obtained  as 
shown  in  equation  (5) . 

PI  = (NLI)(Sit-  QIL)  + (NSI)(3Ii-  (QIS) (SF) ) (5) 

NLI  + NSI 

where : 


PI  is  the  performance  indicator  in  deficiencies  per  page, 

NLI  is  the  number  of  instruments  prepared  and  reviewed  i 00  percent. 


QIL  is  the  average  QI  of  the  100  percent  reviewed  instruments, 
PF  is  the  proficiency  factor  (0  < PF  _<  1.0), 

NSI  is  the  number  of  instruments  prepared  in  the  group  which 


was  sample  reviewed, 

QIS  is  the  average  QI  of  the  sample  reviewed, 

SF  is  the  sampling  factor  10  < SF  <_  1.0). 

If  the  proficiency  factors  vary  between  groups,  or  additional  instrument 
groups  are  necessary,  equation  (5)  can  be  adjusted  accordingly. 

A description  of  the  planned  operation  of  the  QISPI  within  DARC0M  is 
presented  in  Chapter  III. 


CHAPTER  III 
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QUALITY  IMPROVEMENT  SYSTEM  OPERATION 

The  QISPI  discussed  in  Chapter  II  is  presented  in  general  terms,  but 
it  was  developed  for  a specific  application  in  DARCOM.  A system  is  needed 
that  aids  in  the  quality  control  of  solicitation  and  contract  documents 
and  provides  the  required  management  information  at  both  the  commodity  command 
and  HQ  levels.  A description  of  the  envisioned  operation  of  QISPI  at  both 
levels  follows. 

A.  QISPI  OPERATION  AT  COMMODITY  COMMANDS 

All  commodity  commands  presently  employ  some  type  of  procurement  review 
board  and  checklist  to  control  the  quality  of  their  procurement  instruments. 

To  maximize  ease  of  QISPI  implementation,  the  existing  procedures  are  re- 
tained as  much  as  possible. 

1 . Check! ist 

The  present  checklist  concept  is  retained,  but  the  actual  checklists 
are  changed  so  that  all  commodity  coimands  are  using  the  same  standard  to 
gauge  their  documents  against.  Figure  1 shows  the  standard  procurement 
quality  checklist.  The  responsible  buyer  or  PCO  completes  the  first  column 
of  the  checklist  for  every  instrument  prepared  over  $10,000  in  value  and 
places  it  in  the  contract  file  for  future  review.  Even  if  the  instrument  is 
not  formally  reviewed  later,  the  checklist  serves  as  a valuable  guide  during 
instrument  preparation,  because  it  covers  all  substantive  items  and  considers 
both  form  and  content  errors.  A companion  breakout  of  major  and  minor 
deficiencies  is  provided  in  Appendix  A. 
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FIGURE  1.  Procurement  Quality  Checklist. 
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2.  Instrument  Review 


The  present  dollar  threshold  that  must  be  reached  before  a document 
is  reviewed  by  a formal  board  is  $100,000.  Solicitations  and  contracts  less 
than  this  threshold  do  not  receive  a formal  board  review  but  are  reviewed 
only  by  the  buying  activity  responsible  for  the  procurement. 

For  the  QISPI  $100,000  is  used  as  the  threshold  between  100  percent 
review  and  sampled  review.  All  instruments  exceeding  this  threshold  are 
reviewed  by  a formal  board.  It  may  be  feasible  to  increase  this  threshold 
once  the  quality  level  is  determined  and  Is  in  control.  For  those  documents 
less  than  $100,000  but  greater  than  $10,000  (the  small  purchase  threshold), 
a percentage  are  randomly  selected  throughout  the  reporting  period  and  re- 
viewed just  as  the  large  dollar  value  instruments.  The  exact  percentage 
to  be  sampled  is  left  to  the  discretion  of  the  individual  commodity  commands 
since  it  involves  the  expenditure  of  resources  and  affects  their  final  PI 
measurement.  The  results  of  this  review  are  used  to  estimate  the  quality 
level  of  the  smaller  dollar  value  group  for  any  given  reporting  period. 

A board  member  completes  the  second  column  of  the  checklist  during 
document  review.  All  deficiencies  detected  are  recorded  and  narrated  on 
the  back  of  the  checklist.  A board  member  or  assistant  then  records  the 
number  of  document  pages  and  calculates  the  QI  using  equation  (2).  After 
a copy  is  made  of  the  checklist,  it  is  returned  along  with  the  instrument 
and  file  to  the  PC0  responsible  for  correcting  the  detected  deficiencies. 

The  checklist  copy  is  filed  by  the  board  for  future  use. 
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3.  Control  Charts. 


Once  the  QI  is  calculated,  for  an  instrument  or  group  of  instruments, 
it  is  plotted  on  an  appropriate  control  chart  to  determine  if  the  quality 
level  is  acceptable.  One  approach  is  to  maintain  separate  charts  for 
solicitations  and  contracts  and  record  two  variables  on  each.  Each  chart 
would  show  both  the  individual  QIs  for  the  large  dollar  value  document  and 
the  average  QI  for  the  small  dollar  value  document  groups.  Figure  2 illus- 
trates this  approach  for  contracts.  The  specific  values  for  the  means,  UL 
and  U(>,  and  control  limits,  UCL^,  UCL<-,  LCL^  and  LCL<.,  should  be  determined 
from  actual  instrument  quality  data  using  standard  quality  control  procedures. 
Control  limits  will  vary  for  instruments  of  varying  pages. 

4.  Management  Information. 

The  management  information  available  from  the  QI  calculations  and 
control  charts  includes  a quantitative  measure  of  the  procurement  instrument 
quality  level  and  quality  trend  data.  From  this  it  is  possible  to  determine 
if  corrective  action  is  necessary.  If  the  quality  level  and  trend  is  acceptable; 
i.e.,  no  out-of-control  points  or  steadily  increasing  trend  lines,  no 
corrective  action  is  necessary.  Procurement  management  can  be  confident 
that  their  instruments  are  in  control.  If  the  quality  level  or  trend  is  not 
acceptable,  corrective  action  must  be  taken  to  remove  the  causes  of  unaccept- 
able quality.  The  out-of-control  high  points  or  low  points  are  reliable 
indicators  where  corrective  action  is  necessary,  and  the  specific  line  items 
on  the  checklist  for  an  out-of-control  instrument  can  be  used  to  pin-point 
problem  areas.  Corrective  action  includes  such  items  as  personnel  training, 
policy  or  procedure  changes,  increased  allowable  preparation  time,  or,  as 
a last  resort,  personnel  actions. 
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Periodic  quality  information  feedback  at  the  local  level  is  also 
available  and  necessary.  Reports  which  summarize  quality  performance  are 
useful  to  inform  the  buying  organizations  how  their  instruments  compare  with 
other  organizations.  The  reports  also  provide  a convenient  vehicle  for 
disseminating  recent  policy  developments  affecting  instrument  quality  and 
discussing  quality  performance  and  lessons  learned  at  the  other  commodity 
commands  regarding  quality  improvement. 

The  quality  information  available  for  submission  to  the  HQ,  QARCOM  level 
includes  the  following  fourteen  variables: 

NLC  - the  number  of  large  dollar  value  contracts  reviewed 
QLC  - the  average  QI  for  the  large  dollar  value  contracts 
NSC  - the  number  of  small  dollar  value  contracts  prepared  for  award 
QSC  - the  average  QI  of  the  small  dollar  value  contracts  reviewed 
SFC  - the  percentage  of  NSC  that  were  reviewed  before  award 
NLS  - the  number  of  large  dollar  value  solicitations  reviewed 
QLS  - the  average  QI  for  the  large  dollar  value  solicitations 
NSS  - the  number  of  small  dollar  value  solicitations  prepared 
for  issuance 

QSS  - the  average  QI  of  the  small  dollar  value  solicitations 
reviewed 

SFS  - the  percentage  of  NSS  that  were  reviewed  before  issuance 
LCP  - the  number  of  out-of-control  points  for  large  dollar  value 
contracts 
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SCP  - the  number  of  out-of-control  points  for  small  dollar  value 


contracts 

LSP  - the  number  of  out-of-control  points  for  large  dollar  value 
sol icitati ons 

SSP  - the  number  of  out-of-control  points  for  small  dollar  value 
solicitations 

Since  contracts  and  solicitations  cio  not  appear  to  be  statistically  similar 
enough,  a separation  into  two  groups  was  necessary.  This  doubled  the  number 
of  variables  used  to  calculate  the  PI.  A value  for  each  of  these  variables 
is  submitted  to  HQ,  DARCOM  by  each  commodity  command  at  the  end  of  each 
reporting  period. 

B.  QISPI  OPERATION  AT  HQ,  DARCOM 
1 . Performance  Indicator. 

From  the  quality  data  submitted  by  the  commodity  commands,  an 
action  officer  (AO)  at  HQ,  DARCOM  calculates  the  performance  indicator  (PI) 
for  each  using  equation  (6). 


PI  = (NLC)(9^  . QLC)  + (NSC)(^  - QSC-SFC)  + (NLS)(j^§  -QLS)  + (NSS)(^|  - QSS'SFS) 

NLC  + NSC  + NLS  + NSS 


(6) 

where : 

PI  is  the  performance  indicator  in  deficiencies/paqe, 

PFC  is  the  review  board  proficiency  factor  for  contracts, 

PFS  is  the  review  board  proficiency  factor  for  solicitations  for 
each  command,  and  the  other  variables  are  as  defined  previously. 


In  order  for  the  QISPI  to  function  properly,  it  is  critical  that  the  review 


boards  conduct  a thorough  and  consistent  review.  The  degree  to  which  this 
is  accomplished  is  reflected  in  the  proficiency  factors.  The  method  pro- 
posed to  establish  and  maintain  reasonable  estimates  of  their  value  for  each 
coimand  is  through  periodic,  independent  reviews  by  some  organization  such 
as  the  DARCOM  Procurement  Management  Review  (PMR)  organization.  PMR  would 
periodically  update  the  proficiency  factors  through  an  independent  review 
on  a sampling  basis  after  the  board  review.  This  independent  review  would 
tend  to  "keep  the  boards  honest." 

Once  the  PI  values  are  calculated,  they  are  weighted  by  the  applicable 
number  of  instruments  for  each  command  and  averaged  for  the  DARCOM  PI 
measurement  using  equation  (7). 


m 

pid=  V (pqHN,) 

t •• 

where : 

P I D is  the  DARCOM  Performance  Indicator 

is  the  total  number  of  instruments  processed  by  the  respective 
commands  during  the  reporting  period;  (i.e.,  NLC+NSC+NLS+NSS) 
m is  the  number  of  comnodity  conmands  reporting. 
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2.  Management  Information. 


The  DARCOM  AO  prepares  one  summary  report  for  presentation  during 
program  review.  This  routine  report  shows  the  performance  variables  sub- 
mitted by  the  commands  and  the  calculated  Pis  for  comparison  with  previously 
established  targets.  The  report  also  contains  any  narrative  required  to 
explain  achieved  performance.  Figure  3 illustrates  one  possible  viewgraph 
format  for  the  PI  summary  report. 

The  targets  to  be  achieved  should  be  developed  from  actual  instrument 
quality  data  from  the  commodity  commands.  It  is  not  necessary  that  the  targets 
be  the  same  for  all  commands  but  they  must  be  realistic.  Each  command  should 
strive  to  attain  its  target  and  be  judged  accordingly.  Between  command 
comparisons  should  be  avoided. 

Feedback  to  the  commodity  command  from  DARCOM  HQ  could  be  provided  by 
using  a monthly  or  quarterly  Procurement  Information  Newsletter.  Such  a 
newsletter  would  summarize  recent  policy  developments  affecting  document 
quality  and  discuss  quality  performance  and  lessons  learned  at  the  MSCs 
regarding  quality  improvement. 

3.  Performance  Analysis. 

The  DARCOM  AO  compares  the  calculated  Pis  with  the  previously 
established  targets  to  determine  if  the  quality  level  at  the  commands  is 
acceptable.  If  the  PI  values  are  at  or  below  the  targets  and  the  number 
of  out-of-control  points  is  not  excessive,  it  can  be  concluded  that  the 
DARCOM  procurement  instruments  are  at  an  acceptable  quality  level.  There  is 
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no  need  for  additional  analysis  except  in  the  case  that  a particular 
commodity  command  is  showing  exceptional  quality.  It  may  be  that  something 
can  be  learned  from  them  and  applied  to  the  other  commands. 


If  the  calculations  show  missed  targets,  especially  consistently  or 
substantially,  or  an  excessive  number  of  out-of-control  points,  further 
investigation  is  needed  to  determine  the  causes.  Are  the  targets  unrealistic? 
Has  personnel  turbulance  been  a contributing  factor?  Can  the  problem  be 
traced  to  a particular  procurement  group  or  is  it  omnipresent?  Most  of 
these  kinds  of  questions  must  be  answered  by  the  commands  themselves  to  the 
satisfaction  of  the  DARCOM  action  officer. 

Corrective  action  must  be  taken  to  get  the  performance  "back  on  track" 
once  the  causes  are  identified.  This  corrective  action  is  primarily  the 
responsibility  of  the  individual  commands.  Investigation  into  the  other 
indirect  quality  indicators  listed  in  Chapter  II  may  provide  additional 
insight  into  why  the  problem  exists  or  whether  or  not  it  truly  is  a problem. 

Before  the  QISPI  can  be  implemented' in  DARCOM,  a few  remaining  questions 
must  be  answered  satisfactorily  regarding  actual  deficiency  rates,  per- 
formance targets,  and  system  applicability.  The  system  tests  described  in 
Chapter  IV  attempt  to  answer  these  questions. 


CHAPTER  IV 


QUALITY  IMPROVEMENT  SYSTEM  TEST 

Two  different  approaches  are  being  taken  to  test  the  QISPI  as  designed 
in  Chapter  II:  (1)  computer  simulation  and  (2)  test  application  at  two 

commodity  commands.  The  system  simulation  is  complete;  the  test  applications 
are  still  progressing  with  a planned  completion  date  of  December  1976. 

The  primary  purpose  of  both  approaches  is  to  test  whether  or  not  the 
assumptions  made  to  develop  the  system  are  reasonable  considering  the 
operating  environment  at  the  commodity  comnands.  If  the  assumptions  prove 
to  be  reasonable,  procurement  management  can  implement  the  system  and  be 
confident  that  it  is  based  upon  sound  statistical  principles.  If  the 
assumptions  prove  to  be  unreasonable,  modifications  must  be  made  to  the 
system  to  overcome  the  shortcomings  before  implementation.  If  the  operating 
environment  will  not  permit  the  application  of  this  system,  it  should  be 
discarded  in  favor  of  some  alternative  approach. 

Another  reason  for  the  tests  is  to  establish  realistic  targets  for  the 
commodity  commands  to  strive  for.  The  tests  should  provide  reasonable 
starting  points  for  each  command.  As  the  respective  quality  levels  improve 
over  time,  targets  may  need  to  be  adjusted  accordingly. 

A third  reason  for  the  test  applications  is  to  answer  procedural 
questions  such  as  where  in  the  procurement  process  should  the  reviews  be 
made,  exactly  how  should  they  be  done,  how  much  time  should  be  allowed  for 
each,  and  what  should  the  sampling  factor  be  for  the  smaller  do’lar  value 
instruments?  Answers  to  these  questions  and  others  will  permit  a smoother 
system  implementation. 
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A brief  description  of  both  test  approaches  follows. 


1 


A.  COMPUTER  SIMULATION 
1.  Quality  Indicator. 

A computer  program  was  written  which  simulated  procurement  instru- 
ments with  random  numbers  of  pages  and  random  deficiencies  on  each  page. 

A QI  was  calculated  for  each  instrument  using  equation  (2)  with  assured 
values  for  each  parameter.  The  values  for  the  parameters  were  varied  to 
determine  how  sensitive  the  Ql  is  to  each  and  what  the  range  of  validity 
is  for  the  system  assumotions.  All  of  the  assumed  values  in  this  chapter 
are  based  upon  actual  procurement  instrument  data  from  two  DARCOM  commodity 
commands  and  one  Air  Force  Logistics  Center.  Adjustments  have  oeen  made 
where  good  judgement  dictated. 

The  frequency  distribution  plotted  on  Graph  1 for  500  instruments  is 
representative  of  the  QIs  that  result  from  the  calculations  using  the 
current  best  estimates  of  the  equation  (2)  values.  Page  numbers  were  assumed 
to  be  uniformly  distributed  with  a mean  of  60  pages  and  a range  of  20  to 
99  pages.  The  number  of  deficiencies  detected  on  each  page  was  assumed  to 
follow  a Poisson  distribution  with  a mean  of  .05  major  deficiencies  per  page 
and  .10  minor  deficiencies  per  page.  The  weighting  constant  was  set  equal 
to  four.  The  influence  of  the  assimed  Poisson  distribution  is  evident  from 
the  graph.  Since  the  deficiency  parameters  are  small  and  some  of  the  sample 
sizes  (number  of  document  pages)  are  also  small,  this  is  to  be  expected. 
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As  the  deficiency  parameters  increase,  the  distribution  tends  more 
toward  normality.  Graph  2 illustrates  this  for  500  instruments  with  major 
and  minor  deficiency  parameters  equal  to  .10  and  .20  respectively.  All 
other  values  remained  the  same  as  in  Graph  1. 

As  the  deficiency  parameters  decrease,  the  Poisson  influence  becomes 
more  pronounced  and  the  distribution  is  well  described  as  chi-square. 

This  is  illustrated  in  Graph  3 for  500  instruments  with  major  and  minor 
deficiency  parameters  equal  to  .005  and  .01  respectively. 

If  the  deficiency  parameters  are  held  at  .U5  and  .10  respectively  and 
the  number  of  instrument  pages  is  varied,  the  distribution  tends  toward 
normality  as  the  number  of  pages  increases.  Graph  4 with  the  number  of  pages 
equal  to  20  and  Graph  5 with  the  number  of  pages  equal  to  100  show  the 
di fference. 

Graph  6 shows  there  is  little  effect  of  doubling  the  weighting  constant 
to  eight  with  all  other  values  the  same  as  in  Graph  1. 

The  QI  mean  is  not  affected  by  an  increase  in  sample  size,  but  the 
standard  deviation  decreases  considerably.  This  is  shown  in  Table  1.  QIs 
were  generated  for  300  instruments  in  each  group  varying  only  the  number  of 
pages  from  the  data  used  for  Graph  1 . 

Goodness  of  fit  tests  for  normality  for  each  of  the  graphs  shown  are 
postponed  until  after  the  initial  test  application  data  is  available. 
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TABLE  1.  SENSITIVITY  OF  QI  TO  NUMBER  OF  PAGES 


NUMBER  01 
INSTRUMENT 

QI 

QI 

STANDARD 

AVG.  NO. 
OF  MAJOR 

AVG.  NO. 
OF  MINOR 

PAGES 

MEAN 

DEVIATION 

ERRORS 

ERRORS 

20 

0.3032 

0.2144 

0.99 

2.09 

30 

0.3039 

0.1718 

1.50 

3.13 

40 

0.2962 

0.1445 

1.95 

4.05 

50 

0.3031 

0.1331 

2.54 

4.99 

60 

0.3116 

0.1187 

3.17 

6.03 

70 

0.2946 

0.1105 

3.48 

6.71 

80 

0.2985 

0.1019 

3.97 

8.01 

90 

0.2948 

0.0966 

4.37 

9.06 

100 

0.3043 

0.0910 

5.07 

10.16 

110 

0.2965 

0.0888 

5.39 

11.07 

120 

0.3063 

0.0872 

6.12 

12.29 
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2.  Performance  Indicator. 

The  performance  indicator  (PI)  in  this  system  is  a measure  of  the 
number  of  errors  remaining  In  the  instruments  after  preparation  and  review. 

It  is  a weighted  average  of  QIs  adjusted  for  review  board  proficiency  and 
sampling  percentages.  Table  2 gives  the  results  of  simulating  an  average 
commodity  conmand's  activity  for  one  year  using  equation  (5)  as  the  expression 
for  PI. 


TABLE  2. 

SIMULATED  PI 

VALUES 

MONTH 

PERFORMANCE 

INDICATOR 

(PI) 

NO.  OF 
LARGE 
CONTRACTS 
(NLC) 

QI  FOR 
LARGE 
CONTRACTS 
(QLC) 

NO.  OF 
SMALL 
CONTRACTS 
(NSC) 

QI  FOR 
SMALL 
CONTRACTS 
(QSC) 

1 

.1524 

100 

.2885 

100 

.2995 

2 

.1438 

100 

.2832 

100 

.2812 

3 

.1477 

100 

.2998 

100 

.2876 

4 

.1475 

100 

.2968 

100 

.2876 

5 

.1382 

100 

.2905 

100 

.2680 

6 

.1442 

100 

.3055 

100 

.2792 

7 

.1478 

100 

.3087 

100 

.2869 

8 

.1412 

100 

.3001 

100 

.2734 

9 

.1500 

100 

.3105 

100 

.2915 

10 

.1489 

100 

.2993 

100 

.2903 

11 

.1520 

100 

.2976 

100 

.2974 

12 

.1558 

100 

.2879 

100 

.3069 
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The  proficiency  factor  was  set  equal  to  .90,  and  the  sampling  factor 
was  .20.  The  other  data  was  the  same  as  used  in  Graph  1.  The  number  of 
documents  simulated  are  based  upon  estimates  using  data  from  the  "Procurement 
Statistics"  for  FY  75  and  76. ^ 

Since  the  frequency  distribution  for  the  average  of  many  QI  values  is 
approximately  normal  by  the  Central  Limit  Theorem,  the  frequency  distribution 
of  PI  is  also  normal  with  a small  variance.  The  mean  and  standard  deviation 
for  the  PI  data  in  Table  2 is  .1475  and  .0050  respectively. 

B.  TEST  APPLICATIONS 

Two  DARC0M  commodity  commands  tested  the  QISPI  to  determine  its  acceptability 
in  an  actual  operating  environment.  During  the  test  period  they  reviewed  as 
many  contracts  and  solicitations  as  their  resources  permitted  using  the  pro- 
cedure outlined  in  Chapter  III.  These  test  applications  should  answer  most  of 
the  remaining  questions  regarding  QISPI  development  and  identify  where  modifi- 
cations are  required.  Since  one  command  uses  a permanent,  full-time  review 
board  and  the  other  uses  an  ad  hoc  board,  tne  '.est  should  provide  a better 
understanding  of  the  system's  applicability  under  either  approach. 


9 

"Procurement  Statistics".  Procurement  Statistics  Office,  HQDA 
(DAMA-ISP),  Washington,  DC  20310.  Fiscal  years  75  and  76. 
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CHAPTER  V 


CONCLUSION 

The  QISPI,  as  developed  and  tested  to  date,  has  considerable  potential 
as  a tool  for  DARCOM  procurement  managers  to  control  the  quality  of  their 
procurement  instruments.  When  implemented,  it  should  provide  the  meaningful 
management  information  on  document  quality  that  is  presently  lacking  in 
DARCOM.  Without  such  information,  instrument  quality  control  and  improve- 
ment is  at  best  haphazard. 

The  simulation  tests  have  shown  that  the  mathematical  structure  of  the 
QISPI  is  feasible.  The  results  of  the  test  applications  at  two  commodity 
commands  will  be  evaluated  to  determine  the  appropriateness  and  utility  of 
the  system  in  an  operating  environment.  Until  the  system  test  results  are 
evaluated  and  the  remaining  questions  are  answered  satisfactorily,  recommenda- 
tions regarding  system  implementation  will  not  be  made. 

This  report  has  been  published  to  present  the  currently  designed  system 
to  potential  users  and  to  provoke  them  into  thinking  more  about  the  subject 
of  procurement  instrument  quality.  A final  report  will  be  published  following 
the  system  tests.  The  report  will  present  the  test  results  anc  any  required 
system  modifications  as  well  as  recommendations  for  implementation  throughout 
DARCOM. 
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APPENDIX  A 

dfi'inttton  ok  ma.iok/mtnor  frkor^ 


r.KrrmN  a 

1.  Type  <>f  Contruct/Modificntion/Solicitation/Amenclment 

A.  Inappropriate  type  of  contract 

B.  Code  Incorrect 


2 . Con tract /Mod i fi cat ion/Soli ci tat ion/Amendment  Number 

A.  Number  omitted 

B.  Incorrect  number 

3.  Issuing,  Administrative,  Paying  Offices 

A.  Information  omitted 

B.  Incorrect  information 

C.  Incorrect  code 

4.  Delivery,  Discount  and  Invoice  Information 

A.  Delivery/FOB  Point  omitted 

B.  Delivery/FOB  Point  incorrect 

C.  Discount  omitted  or  incorrect 

(if  no  discount  offered  by  Contractor,  ommission 
of  "none"  or  "N/A"  is  not  an  error) 

D.  Omission  of  place  for  submission  of  invoices 


5.  Contractor's  Name  and  Address 

A.  Incorrect  Contractor 

B.  Address  incorrect 

C.  Incorrect  code 


6.  Ship  To/  Mark  For 


A.  Incorrect  destination  or  destination  omitted 

B.  MITjSTRIP  Data  not  consistent  with  that  provided 
to  the  Contract  Specialist,  or  omitted 

C.  Mark  For  information  omitted  or  incorrect 


'(.  Accounting  and  Appropriation  Data 

A.  Adi  pint  1 certified  funds  riot  available 

B.  Ace  anting  and  Appropri  it  ion  data  -ited  incorrectly 

C.  Ac  muting  arid  Appro  print  ion  data  i.nitted 


Major 

Minor 


Minor 

Minor 


Minor 

Minor 

Minor 


Major 

Major 

Major 

Minor 


Major 

Major 

Minor 


Major 

Minor 

Minor 


A-l 


Major 

Major 

Major 


8.  Contract/Modification  Amount  (Also  Section  Ii) 


A.  Contract  price  (unit  and  total,  if  applicable)  omitted 
H.  Amount  cited  not  consistent  with  the  negotiated  agreement, 
or  competitive  price 

C.  Insufficient  justification  for  contract  amount 


9.  Negotiation  Authority 

A.  Authority  cited  incorrectly 

B.  No  negotiation  authority,  if  required 

C.  Negotiation  authority  not  appropriate  or  insufficient 
justification  for  its  use 

10.  Other  Section  A Elements 

A.  If  sole  source,  omission  of  sole  source  justification  or 
NCPSA  and  'proper  signatures/approvals 

B.  Omission  of  Small  Business  review 

C.  Omission  of  DO/DX  Rating 

SECTION  B 

11.  Certifications  and  Representations 

A.  Required  certification  and  Representation  omitted 

B.  Clause  not  in  the  latest  form  required  by  ASPR  or 
other  regulations 

C.  Inappropriate  provision  included 

12.  Contractor's  Submission  of  Certifications  and  Representations 
A.  Certification  and  Representation  not  completed 

SKCTior:  c 

lq.  Instructions,  Conditions,  Notices 

A.  Omission  of  pertinent  information 

B.  Inclusion  of  inappropriate  information 

C.  Ambiguous  information 

SECTION  P 

lit.  Evaluation  Factors 

A.  Pertinent  factors  omitted 

(i.r.  discount,  Government  Production  and  Research  Property, 
K.A.  testing,  transportation) 


Major 

Major 

Major 


Minor 

Major 

Major 


Major 

Major 

Minor 


Minor 

Minor 

Minor 


Minor 


Minor 

Minot- 

Minor 


A-2 


lrj\  Cv- 


vnc*ors  other  them  price 


A.  .Pertinent  factors  omitted  Mnjor 
R.  Relative  order  of  importance  omitted  or  incorrect  Mnjor 
C.  Factor;;  not  conr.ir.1ent  with  Evaluation  Plan  Major 
1>.  Evaluation  weight,  net  forth  in  contract  Minor 


’6.  Other  Section  D Elements 
SECT! Oh’  E 

17.  I.ine  and  Subline  Item  Number 

A.  CLTN/SUBCLIN  (number)  omitted 

B.  OLIN/SURCLIN  (number)  incorrectly  identified 

18.  Nomenclature,  NSN,  Part,  Number,  Quantity 


A.  Omission  of  data  Major 

3.  Incorrect  Nomenclature  Maior 

C.  Incorrect  NSN  or  Part  Number  Maior 

D.  Incorrect  Quantity  Major 


19.  Other  Section  E Elements 
SFQTrrr:  !•’ 

20.  Description  and  Specifications  including  Revisions  and  Dates 

A.  Description  or  Spec  omitted  Major 

B.  Revision/date  emitted  or  unidentified  Major 

C.  Incorrect  description,  specification,  or  revisions  Major 

21.  Brand  Name  or  Equal  Statement 

A.  Statement  omitted 

B.  Statement  incorrect 

22.  Other  Sect  ion  F Elements 
F.CTroN 

Preservation,  Packaging,  Packing,  Marking 

A.  Information  emitted  or  incorrect  Major 


Mnjor 

Major 


Minor 

Minor 


n}-Y--r  f n 

2fi . Delivery  or  Performance  Date 


A.  Information  omitted  or  incorrect 

B.  Place  and  method  ol'  delivery,  if  FOB  destination 

C.  Place  and  method  of  delivery,  if  FOB  origin 

SECTION  I 

?').  Inspection  and  Acceptance  Requirements 

A.  Inspection  and  Acceptance  point  omitted  or  incorrect 


26.  Other  Inspection  or  Acceptance  Requirements  (i.e.  Fly-to-Buy) 
omitted  or  incorrect 

SECTION  .T 

27.  Special  Provisions 

A.  Option,  EPA,  Incentive  arrangement,  F.A.  approval, 
Warranty  Clause,  GFP  and  approvals,  DTC,  Award  Fee 
Provisions  omitted,  incorrect  or  ambiguous 

B.  Special  Provisions  ambiguous  or  inappropriate  or 
propei-  approvals  not  obtained 

C.  Technical  liaison  clause  omitted  or  incorrect 

D.  Other  Special  Provisions 


SECTION  K 

28.  Paying  and  ACO  Instructions  Including  PCO  I if  '■mation 
A.  Information  omitted  or  incorrect 


29.  Other  Section  K Elements 


SECTION  1 

30.  Required  General  Provisions 

A.  Provision  omitted  or  not  current 

31.  Additional  Applicable  General  Provisions 
A.  Provision  omitted  or  not  current 

32.  Appropriate  Alteration  to  General  Provisions 


A- 4 


Major 

Major 

Minor 

Major 

Major 

Major 

Major 

Minor 

Minor 

Minor 

Major 

Major 


A. 


Alteration  omitted  or 


i ncorrect 


Ma  jor 


APPENDIX  B 


STUDY  TEAM  COMPOSITION 

Monte  G.  Norton,  (Project  Officer)  Industrial  Engineer,  US  Army  Pro- 
curement Research  Office,  ALMC.  B.S.  Industrial  Engineering,  North  Dakota 
State  University,  1969;  M.E.  Industrial  Engineering,  Texas  A&M  University, 

1970.  Prior  to  joining  Arrr\y  Procurement  Research  Office,  Mr.  Norton  was  an 
Operations  Research  Analyst  with  the  Defense  Logistics  Studies  Information 
Exchnage  (DLSIE).  Before  that,  Mr.  Norton  was  a General  Engineer  with  the 
Safeguard  System  Command,  Alabama  and  has  been  a Government  subcontractor. 

Frederick  W.  Helwig,  Procurement  Analyst,  US  Army  Procurement  Research 
Office,  ALMC.  B.A.  in  Economics,  University  of  South  Florida,  1963;  Master 
of  Commercial  Science,  Rollins  College,  1970.  Prior  to  joining  the  US  Army 
Procurement  Research  Office,  Mr.  Helwig  was  a Contract  Negotiator  (R&D  and 
Production  Contracts)  with  the  Navy.  He  also  has  had  similar  procurement 
experience  as  a Procurement  Officer  and  Contract  Negotiator  with  the  Air  Force. 

Edward  T.  Lovett,  Procurement  Analyst,  US  Army  Procurement  Research 
Office,  ALMC.  B.A.  in  Psychology,  .illanova  University  1969;  M.S.  in  Con- 
tract and  Procurement  Management,  Florida  Institute  of  Technology,  1974. 

Prior  to  joining  the  US  Army  Procurement  Research  Office,  Mr.  Lovett  was  a 
Logistics  Management  Specialist  with  the  Directorate  for  Requirements  and 
Procurement,  HQ  DARCOM  and  a Contract  Assistant  with  the  US  Army  Electronics 
Command  at  Ft  Monmouth. 


B-l 


UNCLASSIFIED 

Sr  run  t Ci.issificMMor, 

DOCUMENT  CONTROL  DATA  - R & D 

(Secun  y classification  of  titli bet  civ  r/  abstrac  t and  indexing  annotation  ,,.%f  «•  ttervd  when  the  overall  report  Is  classified 

l.  ORIGINATING  ACTIVITY  (Corporate  author)  1 2a.  REPORT  SECURITY  CLASSIFICATION 

US  Army  Procurement  Research  Office,  ALMC  Unclassified 

Fort  Lee,  VA  23801  ^ *t>.  croup 


3.  R fc  PORT  ’ ITLE 

Quality  Improvement  System  for  Procurement  Instruments 

/ 


4.  DESCRIPTIVE  NOTES  ( Typo  of  report  and  Inclusive  dates) 

Interim  Apr  76  - Feb  77 


5-  AuTKORtS)  (First  name,  middle  initial,  last  name) 

Monte  G.  Norton 


0 REPORT  DATE 

Feb  77  J 

la.  TOTAL  NO.  OF  PACES 

58 

lb.  NO  Of  REES 
11 

0*.  CONTRACT  OR  GRANT  NO. 

9a.  ORIGINATOR’S  REPORT  NUMBER!*) 

b.  PROJEC  T NO. 

APRO  613-1  / 

C. 

[oh.  OTHER  REPORT  NO(S)  (Any  other  nu.:  beta  that  may  be  t-a  signed 

this  report) 

d. 

10  DISTKID'JTION  STATEMENT 


Distribution  of  the  document  is  unlimited. 


I SU  P P t l Ml  NT  ARY  NOTLS 


I J TA  US  T (<  A C T 


12.  SPONSORING  MILITARY  ACTIVITY 

USA  Materiel  Development  ft  Readiness 
Command , ATTN:  DRCPP-SP,  5001  Eisenhower 
Avenue,  Alexandria,  VA  22333 


the  Quality  Improvement  System  for  Procurement  Instruments  (QISPI)  as  presented  n 
this  report  has  considerable  potential  as  a tool  for  procurement  managers  to  control 
the  quality  of  their  instruments.  Document  quality  measurement  and  control  begins 
with  a quality  indicator  (Q1 ) calculation  based  upon  the  number  and  type  ol  instru- 
ment deficiencies  per  page.  Standard  control  charts  are  used  to  determine  document 
acceptability  and  track  quality  levels  at  the  commodity  commands.  The  Q1  values 
from  each  command  are  adjusted  at  the  HQ  DARCOM  level  to  ref lect  review  board 
proficiency  and  sampling  percentages  which  results  in  the  final  performance 
indicator  (PI)  calculation  for  all  of  DAr.COM.  System  test  results  from  tests  con 
ducted  at  selected  major  procurement  activities  of  DARCOM  will  be  carefully 
evaluated  during  and  after  the  test  to  confirm  the  appropriateness  and  utility  of 
the  system  and  to  make  any  necessary  improven  mts.  Reco  lendat ions  regan  if 
implementation  are  postponed  until  the  tests  are  co  pleted . . (II) 
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